Computing devices typically include a central processing device (CPU) or other control unit that executes instructions. The instruction may be packaged together to form a computer program or executable. These computer programs or executables are generally referred to as software. The computing devices may include memory or other storage media that store the computer programs and, in response to a user's command, the control unit may load or otherwise execute one or more of the computer programs indicated by the user command. In this respect, computing devices may represent a platform on which a wide variety of software may execute. Example computing devices include general purpose computers (e.g., desktop computers, laptops, and network servers) as well as other devices that contain embedded CPUs or other control units (e.g., routers, printers, switches, firewalls, monitors, disk drives, telephones, television, automobiles or any such computer-controlled device).
To develop software, a team of one or more computer programmers typically writes code in a high-level programming language, such as C, C++, Java, or Visual Basic. The code may comprise a number of functions called by a main executable. This development team, upon completing particular portions, e.g., functions, of the software may compile these portions into machine code or other low-level instructions that are executable by the control unit. This low-level code that is executable by the control unit may represent the above mentioned software or executables.
After compiling the high-level code to generate the software, the development team may proceed to verify the functionality of the software when executed by the target device. Typically, the development team maintains a test environment in which to test this software when deployed on the target device. For example, when the target computing device is a network device to be deployed within a packet-based network, the test environment may include one or more computing devices that can be configured to simulate one or more “test beds.” These test beds may each represent a particular test scenario with respect to emulated network configurations. The development team may design test cases that utilize these test beds to test a particular functionality or behavior of the software when executed by the target network device with respect to the requirements of the particular network configuration
One test case, for example, may test functionality of the software related to establishing a communication session and communicating data by way of the communication session. The development team may manually implement this test case by configuring the computing devices of the test environment such that a first one of the computing devices establishes a communication session with a second one of the computing devices that executes the software to be tested (i.e., the target computing device). The development team may then cause the first computing device to transmit data via the communication session to the second computing device and observe the functionality and behavior of the second computing device as it executes the software to respond to these communications. The development team may report any functional errors observed during this testing and debug the high-level code to resolve the errors. The development team may, after debugging the high-level code, recompile the high-level code into the low-level code, and re-implement the test case to determine whether the functional error has been corrected. The development team may repeat this process until all functional errors have been debugged and, after debugging all of the errors, release the software to the public.
For large-scale software development or software having a large code base, the development team may automate this testing to increase the efficiency with which testing may be accomplished. To automate the testing, the development team commonly implements each test case as a script or other programmable module that the test environment may execute to automatically configure each test bed and programmatically cause the computing devices of the configured test bed to interact with one another in a desired way. These automated test cases may further evaluate the interactions between the computing devices in order to assess a particular functionality or behavior of the software. The automated tests may also report the outcome of the test identifying inappropriate behavior, e.g., functional errors, or other deficiencies of the software. As the automation may not require human intervention, the automation of the test cases may increase testing efficiency.
To ensure each subsequent release of the software is free of software bugs, e.g., functional errors, another team of one or more computer programmers or quality assurance specialists may perform regression testing against the software to be released. This regression testing team may collect each of the automated tests generated prior to each successive release of the software and maintain every one of these automated test cases in a regression test case database. Prior to releasing each version of the software, the regression testing team may run each and every one of the automated tests collected for previous versions of the software to test the new or next version of the software prior to releasing this new version of the software for public use. This form of full regression testing may ensure consistent behavior of the software by verifying that changes made to previous versions of the software in order to generate the new version have not reintroduced functional errors that impacted functionality previously incorporated within older versions of the software.